Computer systems often utilize a cache to improve computing performance and throughput by reducing the apparent time delay or latency normally associated with a processor accessing data in a main memory. Such a computer system may employ one or more caches, each including a cache memory in conjunction with control logic, such as a cache controller. Generally, each of the cache memories is smaller and faster than the main memory, so that a processor may access a copy of data from the cache memory more quickly and readily than from the main memory. To this end, computer systems often employ caches having memories that provide enough access bandwidth to handle the highest memory access rate (i.e. the “demand rate”) of the system processors.
Typically, different types of processor workloads dictate different demand rates. If a cache is not designed to handle the maximum demand rate of its associated processor, many of the requests for access to the cache memory must be queued for some period of time. If the memory requests continue at a high rate, the length of the access queue increases, possibly to a level at which the resulting latency for some of the queued memory accesses is longer than the latency associated with a direct access to the main memory. As a result, for those periods of time, the cache actually lengthens memory access latency, thus becoming a performance hindrance within the computer system.
To prevent such a decrease in performance, caches typically are designed to handle the maximum demand rate, as described above, which often may involve complex cache designs and correspondingly expensive cache memories, due to the high access bandwidth they need to provide. Moreover, in some systems, various physical or design constraints, such as integrated circuit (IC) pinout, printed circuit board (PCB) layout, thermal characteristics, design complexity, time-to-market, and manufacturing costs, may prevent the system designer from implementing a cache providing the necessary bandwidth, thus leaving the designer with no option but to forego the implementation of a cache in the computer system altogether.